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Start saving your time today and find out how usefull skills you can learn with Carlos. 


In this workshop you will see real life situations, where debugging skills will save you time, headaches 
and possibly find a solution with minimal amount of effort. 


Debugging/Troubleshooting is a really useful skill when you are working in maintaining legacy applications, doing some small 
incremental changes to an old code base, where the code has been touched by so many hands over the years and it is 
becoming really a mess. So, management has decided that the code works as it is and you are not allowed to change it all over 
“the right way ™”. 
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: Advanced inspection of data structures and variables 
‘ Introduction to the jdb debugger 

: Working with core dumps in GDB 

: Introduction to Dtrace 


° Course Materials 


About the Instructor 


Carlos Antonio Neira Bustos has worked several years as 
a C/C++ developer and kernel porting and debugging 
enterprise legacy applications. He is currently employed 
as aC developer under Z/OS, debugging and 
troubleshooting legacy applications for a global financial 
company. Also he is engaged in independent research on 
affective computing. 


Course format: 


The course is self-paced — you can visit the training whenever you want and your content will be there. 
Once you’re in, you keep access forever, even when you finish the course. 

There are no deadlines, except for the ones you set for yourself. 

We designed the course so that a student will need about 18 hours of work to complete the training. 


Your time will be filled with reading, videos, and exercises. 


oie. https://osdmag.org/course/application-debugging-and-troubleshooting-2/ 
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Introduction to the GDB 


The Basics 


To be able to inspect more easily a program, we need to have the symbol table available for the program 
we intend to debug. This is accomplished using the —g flag of the compiler we are going to use (we 
could also debug it without the —g flag, but it is really cumbersome sometimes). In our case, we will use 
FreeBSD 10 as the platform and the clang compiler that comes with it. 


After the program has been compiled using the —g flag, we are able to peek inside it using the gdb 
debugger to start a debugging session. All you need to do is type: 


# gdb <program_name> 


And we will see a (gdb) prompt. That means that we are ready to start typing gdb commands. 


“neira@trueos:~ <shop/1 % gdb exampiel 

WNNU gdb 6.1.1 [FreeBSD] 

“opyright 2004 Free Software Foundation, Inc. 

DB is free software, covered by the GNU General Public License, and you are 
welcome to change it and/or distribute copies of it under certain conditions. 


ype “show copying” to see the conditions. 

here 1S absolutely no warranty for GDB. Type “show warranty” for details. 
his GDB was configured as “amd64-marcel-freebsd”... 

(gdb) 


Or if the program we need to debug is currently running, we must type: 
#gdb 
#(gdb) attach <pid of running program> 


Let’s start with some basic commands and inspect a running application. For this example, | have 
selected this application http://freeciv.wikia.com/wiki/Main_Page. 


“Freeciv is a Free and Open-Source empire-building strategy game inspired by the history of human 
civilization. The game commences in prehistory and your mission is to lead your tribe from the Stone 
Age to the Space Age...” 


We will inspect the game structures at runtime with gdb. So, let’s follow these steps: 


Edit /etc/make.conf and add the line WITH_DEBUG=yes (this will not strip your binaries. You will have 
the symbol table and also add the debug flags to the compiler when compiling the sources of your 
ports). 


Install freeciv from ports. 
Start the freeciv server and client (freeciv-server and freeciv-gtk2). 


Join your local game. 


freeciv (on trucos) 


Figure 1. Joined the local game 
Now, we will use our first gdb command: 


# gdb /usr/local/bin/freeciv-server 


reeciv-server 


right 2004 Free Software Foundatlion, Inc 
is free software, covered by the GNU General Public License, and you are 
velcome to change it and/or distribute copies of it under certain conditions. 
ype “show copying” to see the condit S. 
is absolutely no warranty for . Type “show warranty” for details. 
GDB was configured as “amd64-marcel-freebsd”.. 
r 
usr/local/bin/freeciv-server 


LWP 100405] 
Thread 805406400 (LWP 100405 


is the server for Freeciv version 2. 
learn a lot about Freeciv at http://www. freeciv.org 
jer program has player authentication support, but it's currently not in use. 


aS Easy level Al-controlled player (classic) 
as Easy level Al-controlled aver (classic). 
been added as Easy level Al-controlled 
been added as Easy level Al-control (cl ; 
been added as Easy level AlI-controlled ayer (classic). 
accepting new client connections. 


or introductory help, type 


As we don’t know anything about how Freeciv works, we will press CTRL-C, this will interrupt the 
program and we will take from there. For starters, let’s interrupt and see where we are. If we want to 
continue the execution, we type ‘continue’ or ‘c’ 


rogram received signal SIGINT, Interrupt. 
Switching to Thread 805406400 (LWP 100405/Treectiy 
x0000000801dd606a in select () from /11b/11bc.s0.7 


in select () from /lib/libc.s0.7 
in select Q) from /1ib/libthr.so.3 
7, 3 in fc_select (ne7, readfds-Ox7fffffffdebos, writefds-Ox7fftfftffd6e3s, 
‘ ds-Ox7TTtTttrTdsbs, timeout-Ox7ftrrrittds: at netintf.c:126 

0x000000081 a in server_sniff_all_input () at sernet.c:686 

0x0000000800*' 40/7 in srv_running (©) Cc Srv_matn.' 
in Srv_main () ; Ssrv_main.c:277 

1x00000000004026ca in main (argcel, argveOx/fffffftfda5s) 


Here is a screenshot from the client program freeciv-gtk2. We need to join our local game as we are 
going to debug the server. 


version 2.4.3, qué-gtk-2.0 client 


Sart lew Gere ormect to Network Gate 
Vaart Scenaene Gearre onnect to Garni Jon 
Loed Saved Gene PL, Quit 


rogram received signal SIGINT, Interrupt. 
000801dd606a In select () from /11b/libc.so.7 


00000008010d606a In select () Trom hib/itt 

O0O00000R01LaAB7bDO? in select () from 11b/itbthr.so. 3 

000000800dcd203 In Tfc_select (n-7, readfds-Ox7ftttfffd6es, writefds-Ox7ttttttrd6es, 
epttds-Ox7ffttttftdses, timeout-Ox7TfttttTdsds) at netintt.c:126 

00000008008Tas&8a in Server_sniff_all_input () at sernet.c:686 

00000080090T 407 In Ssrv_running ©) at srv_main.c:2317 

00000080090e0a4 In Srv_main () at Srv_main.c:277 

00000000004026ca In main Caraqcel, araqveOx7TTftftttdasgs) at « 

] 
HxOOOOOO / in srv_running ©) at Sfv_main.c:2317 
ff_all_input() == S_E OTHERWISE) 


1c .S0.7 


x xuM eM eM MM MM 


while (server_sni 


urrent language: auto: currently minigal 
(adb) ist 
? ) 


1 
j 


log_debug("sniffingpackets”); 
j}i_turn_done(); /* HACK: don't wait during AI phases * 
erver_sniff_all_inputQ) S_E OTHERWISE) 


rero the timer: (read-out above o 


The #<num> you see are the stackframes, or simply called frames. When your program is started, the 
stack has only one frame, that of the function main. This is called the initial frame or the outermost 
frame. Each time a function is called, a new frame is created. Each time a function returns, frame for 
that function invocation is eliminated. If a function is recursive, there could be many frames for the same 
function. The frame for the function in which execution is actually occurring is called the innermost 
frame. This is the most recently created of all the stack frames that still exist. 


Let’s go into frame 3. To do this, we type either ‘frame 3’ or ‘f 3’. 


conn_list_iterate(game.all_connections, pconn) { 


if (srvarg.auth_enabled 
&& !pconn->server.is_closing 
&& pconn->server.status != AS ESTABLISHED) { 
auth_process_ status(pconn); 
} 
} conn_list_iterate_end 


if (S_S RUNNING «= server_state() && game.info.timeout «= 
(void) send_server_info_to_metaserver(META_REFRESH); 
return S_E_END_OF_TURN_TIMEOUT; 


con_prompt_o 
if (fc_select(max_desc + 1, &readfs, &writefs, &exceptfs, &tv) == @) 


(void) send_server_info_to_metaserver(META_REFRESH); 
if (game.info.timeout > 
&& S_S RUNNING == server_state() 
&& game.server.phase_timer 
&& (read_timer_seconds(game.server.phase_timer ) 
> game.info.seconds to phasedone)) { 
con_prompt_off(); 
return S E END OF_TURN_TIMEQUT; 


y 
> 
- 


It seems that the server will send us end of turn. Let’s make sure a break point is set as follows: 
the format is > 


<break|b> <source.c>:<line number> (gdb) b sernet.c:695 


log_debug(“sniffingpackets"); 
check_for_full_turn_done(); /* HACK: don't wait during AI phases * 
while (server_sniff_all_input() == S_E_OTHERWISE) { 

nothing * 


' After sniff, re-zero the timer: (read-out above on next loop) 


0x00000008008fa58a in server_sniff_all_input © at sernet.c:686 
5 if (fc_select(max_dedc + 1, &readfs, awritefs, &exceptfs, &tv) == 0) { 


a 


list 


log_debug("sniffingpackets”) ; 
check_for_full_turn_done(); /* HACK: don't wait during AI phases * 
while (server_sniff_all_input() -- S_E_OTHERWISE) { 

* nothing * 


“ After smiff, re-zero the timer: (read-out above on next loop) * 


> 
686 if (fc_select(max_desc + 1, &readfs, a&writefs, &exceptfs, &tv) == 0) { 
(gdb) b sernet.c:695 
sreakpoint 1 at Ox8008fa617: file sernet.c, line 695. 
(gdb) fl 


It seems we are wrong. Let’s interrupt again and inspect the data at this point. 


(gdb) f 3 

ee if (fc_select(max_desc + 1, &readfs, awritefs, &exceptfs, &tv) == 0) 
(gdb) 1 lo 

ast_noplayers = 0 

onnections = false 


- 256 


= § 


ax_desc = 6 

*xcepting = false 

eadfs = {_fds_bits = {89, 0 <repeats 15 times>}} 
writefs = {_fds_bits = {0 <repeats 16 times>}} 
xceptfs - {__fds_bits - {88, 0 <repeats 15 times>}} 
v = {tv_sec = 1, tv_usec = 0} 


Typing i lo means info locals which will display all local variables in this frame and their values, which is 
pretty handy. 


Let’s take a look at something easier to see. Sometimes in freeciv, another civilization will try to enter 
into negotiating terms with us. Therefore, looking at the source code, we find the add_clause function in 
the diptreaty.c source code. That function will add a term which will make the other part accept or reject 
our terms. 


/usr/home/cneira/workshop/freeciv-2.4.3/common 


+~G1ptreaty.¢ 7 
Tex 

j 

| 


Add clause to treaty. 


boo! add_clause(struct Treaty “ptreaty, struct player “ptfrom, 
enum clause_type type, int val) 


A137 struct player “pto (pfrom ptreaty->piro 
Se 138 ? ptreaty->pirl : ptreaty->plro);: 
‘ struct Clause “*pclause;: 
enum diplistate_type ds 
player_dipIistate_get(ptreaty->piro, ptreaty->pir1)->type; 


1F (type < O type > CLAUSE LAST) { 
log_errorC"Illegal clause type encountered.”) 
return FALSE: 


CLAUSE ADVANCE && '!valid_advance_by_number(val)) { 
freebsd-th Thread 8054064 In: add_clause Line: 138 PC: Ox800cT9436 
wratetTs Tds_bits tO <repeats 16 times>} 
axceptfs — {__fds_bits = {88, 0 <repeats 15 times>}} 
v t{tv_sec Ll. tTyv_usec 0} 
(aqdb)> b add_clause 
Breakpoint 1 at Oxs800cT9436 file diptreaty.c, 
(gdb) 
ontinuing. 


treakpoint 1, add_clause (ptreaty-0x8064b9ee0, pfrom-0x8063dd400, type-CLAUSE_CEASEFIRE 
at diptreaty.c:138 


(qdb) § 


After playing a few minutes, we hit a break point. At this point, we don’t even know which civilization has 
approached us to negotiate terms. 


Now, we could know ahead since we set the breakpoint where the negotiation starts. 


bool add_clause(struct Treaty “*ptreaty, struct player “pfroa, 
enum clause_type type, int val) 


struct Clause “pclause; 
enum dipistate_type ds 
player_dipIistate_get(ptreaty->plr0, ptreaty->pirl1)->type; 


if (type < 0 type >= CLAUSE_LAST) { 
log_error("Illegal clause type encountered.”): 


‘eebsd-th Thread 8054064 In: add_clause Line: 138 PC: Ox800cf9436 
vec “SB: +. Skill_tevel AT _LEVEL_EASY, ZZ\ 300, expand 10, science_cost 100, warmth 
frost 0, barbarian_type NOT_A_BARBARIAN, love {1 <repeats 128 times>}}, ai Ox8010891b0, 
was_created «= false, 1S_connected « true, current_conn « Ox0, connections = 0x80682c120, 
gives_shared_vision {vec *\O' <repeats 15 times>}, wonders {O <repeats 200 times> 
attribute_block - {data « Ox0, length =< O}, attmibute_block_buffer - {data - 0x0, length - 
tile_known bits 3888, vec 0x805643400 “"}, rab 0x8064bD92c0, {server (status fyvec ‘\OO1"}, 


got_first_city - false, private_map - 0x8064d0000, really_gives_visi: 
vec ‘\O" <repeats 15 times>}, ivec m}, adv 0x80543 0, {Ox80682d000, Oxo, 


Ox0}, delegate_to - *\O0" <r ats 47 times>, orig_username - '\0 


F client 
v1s10n {t{bits » vec 0Ox8064d0000 “"}, {bits 0, vec 


10 


| assume the negotiation civilization should be in the pfrom pointer. 


gdb) p pfrom 
10 = (struct player *) 0x8063dd400 


gdb) §j 


To print the variables values, we just type ‘p’. In this case, ‘p’ is a pointer to a player structure. If we 
would like to check the definition of the player structure, we just type ptype pfrom, and the structure 
definition will be displayed. 


De = C 
Struct player_slot “slot; 

char name[48]; 

char username[48]; 

char ranked_username[48]; 

int user_turns; 

_Bool 1s_male; 

struct government “government; 

Struct government *target_government; 

struct nation_type *nation; 

--Type <return> to continue, or q <return> to quit---§ 


Now, let’s find out the values of these fields that the civilization demands. Since the pfrom is a pointer, 
we need to use pointer notation to check its contents. 


warmth = 0, Trost = 0, JAP lan_type WIA BARBARLAN ove Lv 


1 <repeats 123 times>}}, al = 0x8010891b0, was_created = false, 1S_connected = false, 
current_conn = 0x0, connections = 0x80682c3e0, gives_shared_vision = {vec = '\0' <repeats 15 times}, 
wonders = {0 <repeats 21 times>, 129, 0 <repeats 178 times>}, attribute_block = {data = 0x0, 
length = 0}, attribute_block_buffer - {data =~ 0x0, length = 0}, tile_known = {bits = 3888, 
vec = 0x805643800 ""}, rgb = Ox8064alea0, {server = {status = {vec = "\001"}, got_first_city = true, 
private_map = 0x80652c000, really_gives_vision = {vec = '\0' <repeats 15 times>}, debug = { 
vec = ""}, adv = 0x80543d800, ais = {0x80626T000, Ox0, Ox0}, 
delegate_to = '\0' <repeats 47 times>, orig_username = ' 
tile_vision = {{bits = 257, vec = 0x80652c000 “"}, {bits = 0, vec = Ox0}} 
gdb) p *pfromll 


is_ 


UxX5U540/410, name Oy Jenki : U" <repea 
0O' <repeats 


username = “Unassigned”, 
ranked_username = “Unassigned”, ‘\0"° <repeats 37 times>, user_turns = 10, 1S_male = true, 
government = 0x805575500, target_government = 0x0, nation = 0x8068470C8, team = 0x805603fe0, 
ady = false, phase_done = false, nturns_idle = 9, 1S_alive = true, revolution_finishes = -1, 
mbassy = {vec = '\O" <repeats 15 times>}, diplstates = 0x80540c400, city_style = 0, 
cities = 0x806 D, units = 0x80682c3c0, score = {happy = 0, content - 2, unhappy - 0, angry = 0, 


specialists = {0 <repeats 20 times>}, wonders = 0, techs = 0, techout = 2, landarea = 

settiedarea - 4000, population - 20, cities - 2, units = 0, pollution - 0, literacy - 0, bnp - 4, 

mfg = 5, spaceship 0, units_built 0, units_killed = 0, units_lost 0, game 2}, economic [ 
-Type <return> to continue, or q <return> to quit-- a] 


Looking at the player struct , it seems that the leader’s name is Roy Jenkins, and looking at the 
backtrace (bt), the clause of the treaty seems to be “cease fire”. So, we are going to be offered a peace 
treaty. 


To continue executing the program, type ‘next’ or ‘n’ . Something like this will be displayed in the 
diplomacy tab: 


a 


Game Edt View Select Unit Work Combat Civilization Help 


Templars European 
Master Jacques de Molay 
Ternplars 


Gold: “dp Add Clause... 
Population: 0 _ Europeans 
Year: 3550 BCE 
Gold: 100 (+0) ES Chief Roy Jenkins 
Tax: 40 Lux: 0 Ses: 60 eae o 7 a paid - 
WII IIIS causes: = 
ro 


Turn Done 
Legion 


Moves: 1 
Jungle 


Accept treaty — }f Cancel meeting 


What you cannot see in the screenshot is that | have requested an embassy in return for the cease-fire 
treaty, but here it is: 


Line: 143 PC: Ox800cfS 


f1d5 in srv_running () at srv_main.c:2261 

e0a4 in Ssrv_main (©) at srv_main.c:2777 
21 0x00000000004026ca In main Cargcel, argveOx7ffffffftdass) at civserver.c:453 
(adb) « 
Continuing. 


Game saved as Treeciv-T0009-Y-3550-auto. Sav.bz2 


Breakpoint 2, add_clause (ptreaty-O0x8064b9ee0, pfrom-0x8063dd400, type-CLAUSE_EMBASSY, val-0) 
at diptreaty.c:13 I 
(gdb) 


Let’s go line by line using next. You could also use the step command but if you use the comman<d, it will 


take you inside a function call instead of just evaluating the function and returning like the next 
command. 


12 


? ptreaty->p : ptreaty->pird); 

struct Clause *nocla se- 

enum dipistate_type ds 
- player_dipIistate_get(ptreaty->piro, ptreaty->piri)->type; 


+--diptreaty.c-- 


og_errorc” 
return FALSE; 


(type «— CLAUSE_ADVANCE 4& 'valid_advance_b unber(val)) 
log_error("Illegal tech value %1 In clause.”, val): 
return FALSE; 


ype) 
a) E && type CLAUSE PE ACE } 
S_ARMISTICE && type ( E_ PEACE) 
YS ALLIANCE && type <= Cl ALL TANCE ) 
YS_CEASEFIRE G& type CLAUSE _CEASEFPIRE))) 


Line: 143 PC: O0x800cT9484 


clause (ptreaty-Ox8064b9ee0, pfrom-0x8063dd400, type-CLAUSE_EMBASSY, val at diptreaty.c:141 
} 
11st 


1 1st 

pt : 

CLAUSE 

ptype CLAUSE 

enum clause ) {CLAUSE ADVANCE, CLAUSE GOLD, CLAUSE MAP, CLAUSE SEAMAP, CLAUSE 


AUSE CEASEFIRE, CLAUSE PEACE, CLAUSE ALLIANCE, CLAUSE VISION, CLAUSE EMBASSY, 


) 
} 
) 
} 


nation_rule_name(nation_of_player(ptreaty->piro)), 
nation_rule_name(nation_of_player(ptreaty->plrl))): 
return FALSE; 


we already have embas 
perror("Illegal embas clause: %S already have embassy with 
nation_rule_name(nation player(pto)), 
nation_rule_name(nation player(pfrom))); 
return FALSE: 


if (!'qame.info.trading_gold && type 
return FALSE: 


if ('game.info.trading_tech && type -— CLAUSE ADVANC 
return FALSE; 


if (!'gqame.info.trading_city && type CLAUSE CITY) 


We are currently at line 143. | just checked what kind of data type was CLAUSE_EMBASSY. It was an 
enum one (Somewhat obvious). 


Using ‘next’ command a couple of times will get us here: 


Keep on typing ‘n’ and we will exit from the function call and arrive to 
handle_diplomacy_create_clause_req 


diptomacy_create_clause_req | counterpart=2, giver=2, typ@=CLAUSE_EMBASSY, 
Value-0) at dipihand.c:679 
88c84d in server_handle_packet pe=PACKET_DIPLOMACY_CREATE_CLAUSE_ REQ, 
8d0250, pplayer-0x8063dbc00, 30) at hand_gen.c: 
b0o0000s0090cD1bD in server_packet_input ( 80, packet=«0x8068d0250, type=-99) 


at srv le 


In incoming _client_packets (pconn-0x800c37580) at sernet.c:460 
in server_sniff_all_input © at sernet.c 
srv_running () at srv_matn. 17 
1 srv_main (€) at srv_main.c:2777 


' d » 7 
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As we keep on typing ‘next’ , we will arrive at this function call_treaty_evaluate that seems interesting. 
Maybe here the results of rejection or acceptance of conditions are done. As | explained earlier, we can 
step into this one using the step command. 


player_number(pother), giver, type, 

Value); 
disend_packet_diplomacy_create_clause(pother->connections, 

player_number(pplayer), giver, type, 

Value); 

treaty); 
call_treaty_evaluate(pother, pplayer, ptreaty); 


Cancel meeting. No sanity checking of input parameters, so don't 
this with input directly from untrusted source. 


Static void really_diplomacy_cancel_meeting(struct player “pplayer, 
struct player “pother) 


struct Treaty “ptreaty = find_treaty(pplayer, pother): 


if (ptreaty) { 
;_ create _clause_re Line: 692 PC: 0x80087d537 
<return> to quit 
argv-Ox7fffffffda5s) at civserver.c:453 


gdb) 
gdb) 
gdb) § 


* FIXME: Should this be put in a ruleset Somewhere? 
#define TURNS_LEFT 16 


such 15 set for 


Static void call_treaty_evaluate(struct player “pplayer, struct player “aplayer, 
struct Treaty “ptreaty) 
(pplayer->ai_controlled) { 
CALL_PLR_AT_FUNC(treaty_evaluate, pplayer, pplayer, aplayer, ptreaty); 


| 
J 


SHAS AHEEEREHHEEEEHHEEEEEHEEEEREHREEE OEHHA OES 


Static void call_treaty_accepted(struct player “pplayer, struct player “*aplayer, 
struct Treaty “ptreaty) 


PC: 0x80087d314 


' list I 

gdb) step 

all_treaty_evaluate (pplayer=0x8063dbc00, aplayer=0x8063dd400, ptreaty=-0x8064b9ee0) at diplhand.c:75 
gdb) § 


Let’s step all the way to get to another point in the program execution. After a couple of steps, we get to 
this point: 


eaty being considered. It 1s all a question about money 


ty_evaluate(struct player “pplayer, struct player “aplayer, 
Struct Treaty *ptreaty) 


bool only_gifts TRUE; 

enum dipistate_type ds_after - “ 
player_dipIistate_get(pplayer, aplayer)->type; 

int given_cities - 0; 


clause_list_iterate(ptreaty->clauses, pclause) 
if (is_pact_clause(pc se->type)) 
ds_after — pact_clause_to_dip!state_type(pclause->type) ; 
1 


if (pclause->type —- CLAUSE_CITY 4&& pclause->from -- pplayer) { 
given_cities++; 


+} cClause_list_iterate_end; 


‘eebsd-th Thread 8054064 In: dai_treaty_evaluate Line: $78 PC: 0x80094a937 


Value=0) at dipihand.c:693 


> (pplayer-0x8063dd400, aplayer-Ox8063dbc00, ptreaty-0x8064b9ee0) at diplihand.c:75 


evaluate (pplayer=0x8063dd400, aplayer«O0x8063dbc00, ptreaty-0x8064b9ee0) at advdiplomacy.c:57 


So, a quick glance at the source code tells us that the total_balance variable is somewhat important to 
evaluate if a clause is accepted (In our case, we are requesting to give us an embassy). Instead of 
printing this variable multiple times, let's leave it available on the display. 


#(gdb) display total balance 
Then, we set a breakpoint somewhere ahead of advdiplomacy.c:621. We can see that the total_balance 


value is displayed and it is -450, seems bad for our proposal. 
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+--a0VG1p |Omacy.C 


b+ #62 if (given_cities 
&& city_list ze(pplayer->cities) given_cities 
always keep at least two cities 
DIPLO_LOG(LOG_DIPL2, pplayer, aplayer, “cannot give last cities"); 
return; 


if (total_balance >. 0) 


handie_diplomacy accept treaty_req(pplayer, player_number(aplayer)); 
DIPLO_LLOG(LOG_DIPL2, pplayer, aplayer, “balance was good: %d", 
total_balance); 


2 the proposal. 
if (pplayer '=j ptreaty->piro) { 
notify ayer, _("*%s (AI)* This deal was not very good for 
name(pplayer), 
er_name(aplaver))-: 


freebsd-th Thread 8054064 In: dai_treaty_evaluate Line: 621 PC: 0x80094ac13 
l: total_balance - 0 

(gdb) list 

(gdb) b advdipjomacy.c:621 

Breakpoint 3 at 0x80094ac13: file advdiplomacy.c, 
(gdb) « 

Continuing. 


line 621. 


Breakpoint 3, dai_treaty_evaluate (pplayer-O0x8063dd400, aplayer-Ox8063dbc00, ptreaty-0x8064b9ee0) 
at advdiplomacy.< 

1: total_balance 

(qdb) 0 


As we can see, total_balance >=0 is the condition to approve the proposal. The following is a resume 
of the commands used in this session: 
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COMMAND ABBREVIATED FORM | WHAT IT DOES 

info local ilo Prints values and names of all local variables in 
the current 
scope. 

backtrace bt A backtrace is a summary of how your program 


got where it is. It shows one line per frame, for 
many frames, starting with the current executing 
frame (frame zero), 

followed by its caller (frame one), and on up the 
stack. 


frame <frame number>|f <frame number> 


The call stack is divided up into contiguous pieces 
called stack frames, or frames for short; each frame 
is the data associated with one call to one function. 
The frame contains the arguments given to the 
function, the function’s local variables, and the 
address at which the function is executing. 


print <variable> 


p <variable> 


displays the value of the variable 


display <variable> 


disp <variable> 


Will automatically print the value of the variable 
being displayed as long as it is within the scope 


win Win Will enter gdb in tui (text user interface) mode if 
we had not entered in the first place. Default 
layout is source at the top commands at the 
bottom. 
next n Executes next line of code. Will not enter 
functions. You can use as parameter the number 
n <number of next to | O/ times to execute next 
perform> 
step Ss Step to next line of code. Will step into a 
s <number of function. 
steps to 
perform> 
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These are really basic commands, but useful. Armed with these commands, you should be able to 
answer a couple of questions using gdb and not just browsing the source code: 


Describe the flow when a city is created, which are the stack frames? 
What is the name of the structure which handles units? 


What is the name of the structure that handles game information? What are the contents of this structure 
at the end of a game? 
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Advanced Inspection of 
Data Structures and 
Variables 


Automating Information Display 


Now that we have used the display command or the print command, it is getting pretty tedious to inspect 
a variable or data structure manually by typing p or display every time we hit a breakpoint we have set. 
There is a command called commands to save us from all this typing. 


First, we set a breakpoint where we want to inspect data automatically. In this case, I’ll check one of the 
city functions. 


(gdb) b city.c:2352 


(gdb) 4 breakpoint keep y 0x0000000800cf1lb7b in citizen base mood at 
city.c:2352 


Now, we can type the following: commands <breakpoint number> 
(gdb) commands 4 


Type commands when breakpoint 4 is hit, one per line. Then end with a line 
Saying "end". 


> 


After you have set the instructions to be executed after the breakpoint is hit, you could modify them or 
just erase them like this: 


(gdb) commands 4 


Type commands when breakpoint 4 is hit, one per line. End with a line saying 
W end" 7 


> end 
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Now, if you want to execute something: 

(gdb) commands 4 

Type commands when breakpoint 4 is hit, one per line. End with a line saying "end". 
> printf “Setting city mood for leader: %s”, pplayer->name 

> end 


We can type all the instructions we want to be executed when this breakpoint is hit. Usually, we used 
print to display values, but there is a more powerful function called printf that uses similar format as the 
C-language function 


(gdb) printf “%Ss”, pplayer->name 


As in C’s printf, ordinary characters in the template are printed verbatim, while conversion specification 
introduced by the ‘%’ character causes subsequent expressions to be evaluated, their values converted 
and formatted according to type and style information encoded in the conversion specifications, and then 
printed. 


For example, you can print two values in hex like this: printf "foo, bar-foo = Ox%x, 0x%x\n", foo, bar-foo 


printf supports all the standard C conversion specifications, including the flags and modifiers between 
the ‘%’ character and the conversion letter, with the following exceptions: 


« The argument-ordering modifiers, such as ‘2$’, are not supported. The modifier “’ is not supported for 
specifying precision or width. 


- The “’ flag (for separation of digits into groups according to LC_NUMERIC’) is not supported. The type 


b) ary ‘ 


modifiers ‘hh’, ‘j, ‘t, and ‘z’ are not supported. 


- The conversion letter ‘n’ (as in ‘Yn’) is not supported. The conversion letters ‘a’ and ‘A’ are not 
supported. 


Note that the ‘Il’ type modifier is supported only if the underlying C implementation used to build GDB 
supports the long long int type. Likewise, the ‘L’ type modifier is supported only if long double type is 
available. 


As in C, printf supports simple backslash-escape sequences such as \n, ‘\t, ‘\V, ‘\'", ‘\a’, and ‘\f, that 
consist of backslash followed by a single character. Octal and hexadecimal escape sequences are not 
supported. 


Additionally, printf supports conversion specifications for DFP (Decimal Floating Point) types using the 
following length modifiers together with a floating point specifier. letters: 


‘H’ for printing Decimal32 types. ‘D’ for printing Decimal64 types. 


‘DD’ for printing Decimal128 types. 
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If the underlying C implementation used to build GDB supports the three length modifiers for DFP types, 
other modifiers such as width and precision will also be available for GDB to use. 


In case there is no such C support, no additional modifiers will be available, and the value will be printed 
in the standard way. 


Here’s an example of printing DFP types using the above conversion letters: printf "D32: %Hf - D64: %Df 
- D128: %DDf\n",1.2345df,1.2E10dd,1.2E1dl 


Dynamically allocated arrays 


Sometimes we will need to take a look at the contents of dynamically allocated arrays (the ones created 
by malloc and calloc system calls). 


For example, we have the usual static memory array: 

char t[8001]; 

It’s easy to display its contents using: 

(gdb) pt 

But what about this one: 

ant. *t; 

t = (int *) malloc ( 8001 * sizeof( int) ); (gdb) pt 

This will give only the address: 

(gdb) p *t 

This will give you the data of the first element in the array, so what is the solution? 
(gdb) p *t@25 

This command will print 25 elements from the array t .The format is pointer@<number of elements. 
Getting information from the symbol table 


When we compiled our program with the —g flag, we instructed the compiler to generate a symbol table 
in our program binary. The table contains variable names, function names and types. Now, let’s suppose 
we want to know the names of all the functions available, we could use one of the info family commands: 


(gdb) info functions 


This command will print the names and data types of all defined functions. If we want to check only the 
function names matching a regexp ,we use the command: info functions <regexp> 
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For example: 


(gdb) info functions city 


The above command matches all functions that have city string in their name. You must use grep regexp 
not perl’s regexp . The same goes with variables with the command: 


(gdb) info variables 


Prints the names and data types of all variables that are declared outside of functions (not the local 
variables). 


Also, the same syntax for info variables regexp 
(gdb) info variables city 


Prints the names and data types of all variables (except for local variables) whose names contain a 
match for regular expression regexp. 


(gdb) info address symbol 


Describes where the data for symbol is stored. For a register variable, this says which register it is kept 
in. For a non-register local variable, this prints the stack-frame offset at which the variable is always 
stored. Note the contrast with ‘print &symbol', which does not work at all for a register variable. For a 
stack local variable, it prints the exact address of the current instantiation of the variable. 


(gdb) whatis exp 


Prints the data type of expression exp. exp is not actually evaluated, and any side-effecting operations 
(such as assignments or function calls) inside it do not take place. Any kind of constant, variable or 
operator defined by the programming language you are using is valid in an expression in GDB. 


(gdb) whatis 
Prints the data type of $, the last value in the value history. 
(gdb) ptype typename 


Prints a description of data type typename. typename may be the name of a type, or for C code it may 
have the form ‘class class-name’, ‘struct struct-tag’, “union union-tag' or “enum enum-tag’. 


(gdb) ptype exp ptype 


Prints a description of the type of expression exp. ptype differs from whatis by printing a detailed 
description, instead of just the name of the type. For example, for this variable declaration: 


struct example {double dtype; float ftype} exl; 


The two commands give this output: 
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(gdb) whatis exl type = struct example (gdb) ptype exl 
type = struct example { double dtype;, float ftype; 


} 


As with whatis, using ptype without an argument refers to the type of $, the last value in the value 
history. 


(gdb) info types regexp 


Prints a brief description of all tyoes whose name matches regexp (or all types in your program, if you 
supply no argument). Each complete typename is matched as though it were a complete line; thus, *i 
type value’ gives information on all types in your program whose name includes the string value. But | 
type “value$' only gives information on types whose complete name is value. This command differs from 
ptype in two ways: first, like whatis, it does not print a detailed description; second, it lists all source files 
where a type is defined. 


(gdb) info source 


Shows the name of the current source file--that is, the source file for the function containing the current 
point of execution--and the language it was written in. (gdb) info sources 


Prints the names of all source files in your program for which there is debugging information, organized 
into two lists: files whose symbols have already been read, and files whose symbols will be read when 
needed. 


(gdb) info functions 
Prints the names and data types of all defined functions. 
(gdb) info functions regexp 


Prints the names and data types of all defined functions whose names contain a match for regular 
expression regexp. Thus, ‘info fun step’ finds all functions whose names include step; 


‘info fun “step' finds those whose names start with step. 
(gdb) info variables 


Prints the names and data types of all variables that are declared outside of functions (i.e., excluding 
local variables). 


(gdb) info variables regexp 


Prints the names and data types of all variables (except for local variables) whose names contain a 
match for regular expression regexp. 


Hey! Stop and look around 
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In GDB, we have three ways of interrupting the program flow and inspect what we need. We have 
Breakpoints, watchpoints and catch points. 


A breakpoint stops the execution at a particular location within the program. We have temporary 
breakpoints, regexp breakpoints and we could set conditional breakpoints. 


The usual breakpoint: 
(gdb) break <source>:<line> 


(gdb) break <source.c>:<function> 


(gdb) break 3 < This one stops at line 3 of the current source file being executed. 

(gdb) break <function> 

The temporary break point is a simple breakpoint that is deleted after it is hit. The command for this is: 
(gdb) tbreak <same format as breakpoint> 

The regexp breakpoint sets breakpoints at the functions matching the regexp provided 

(gdb) rbreak “city 

Conditional breakpoint, stops the execution of the program only if the condition is met. 

(gdb) b if strcmp(commands[0].synopsis,"*start") == 


Yes, you could use the C library functions as long as your program is linked against libc. You can enable 
or disable breakpoints with the following command: 


enable once -- Enable breakpoints for one hit 
enable delete -- Enable breakpoints and deletes when hit 
(gdb) enable once 1 (gdb) enable delete 1 


A watchpoint stops the execution when a particular memory location (or an expression involving one or 
more locations) changes value. Depending on your system, watchpoints may be implemented in 
software or hardware. GDB does software watchpointing by single-stepping your program and testing 
the variable's value each time, which is hundreds of times slower than normal execution. However, it’s 
really useful if you don’t have a clue of where the problem is in your program. 


The syntax for this command is: watch <expr> 
(gdb) watch commands[0] Watchpoint 1: commands [0] 


A catchpoint stops the execution when a particular event occurs. The event could be one of the 
following: 
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Raised signals may be caught: 

catch signal - all signals 

catch signal <signame> - a particular signal 
Raised exceptions may be caught: 

catch throw- all exceptions, when thrown 


catch throw <exceptname> - a particular exception, when thrown catch catch - all exceptions, when 
caught 


catch catch <exceptname> - a particular exception, when caught 
Thread or process events may be caught: 


catch thread_ start - any threads, just after creation catch thread_exit - any threads, just 
before expiration catch thread_join- any threads, just after joins 


Process events may be caught: 


catch start - any processes, just after creation catch exit - any processes, just before expiration 
catch fork - calls to fork() 


catch vfork - calls to vfork() 

catch exec - calls to exec() 

Dynamically-linked library events may be caught: 

catch load - loads of any library 

catch load <libname> _ - loads of a particular library catch unload - unloads of any library 
catch unload <libname> - unloads of a particular library 

The act of your program's execution stopping may also be caught: 

catch stop 

C++ exceptions may be caught: 

catch throw - all exceptions, when thrown catch catch - all exceptions, when caught 


You can enable and delete breakpoints, watchpoints and catchpoints with the enable and delete 
command. 


Proposed Exercises: 


Set a breakpoint that gets triggered when the size of a city is greater than 20 citizens. 
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Do you know what functions modify the game data type? Find out using a watchpoint. 
Display leader’s and city names when a city’s data type is being read or modified by the program. 
When is a government during a revolution being set? Use a watchpoint or a conditional breakpoint. 


For any questions or issues, just use the forum 
http://bsdmag.org/forums/forum/application-debugging-and-troubleshooting/ 


Have fun. 
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Introduction to The JDB 
The Java Debugger 


“The Java Debugger (JDB) is a simple command-line debugger for Java classes. The jdb command and 
its options call the JDB. The jdb command demonstrates the Java Platform Debugger Architecture 
(JDBA) and provides inspection and debugging of a local or remote Java Virtual Machine (JVM). See 
Java Platform Debugger Architecture (JDBA) at 
http://docs.oracle.com/javase/8/docs/technotes/guides/jpda/index.html” 


JDB, like GDB, is a command line debug tool that will help us find bugs in our code (it is provided by 
openjdk), hopefully with little effort. The application that will be used to demonstrate the usefulness of 
JDB will follow the setting as the last one. The application we are going to use is called Freecol: 


"The FreeCol team aims to create an Open-Source version of Colonization (released under the 
GPL). At first, we'll try to make an exact clone of Colonization. " 


Like the last time we used GDB, we need the Freecol application sources to be compiled with extra 
debugging symbols (-g flag to the java compiler). 


Requirements: You must download the following package : 
http://prdownloads.sourceforge.net/freecol/freecol- 0.10.7-src.zip?download 


You must have the openjdk7 package installed 3- You must have apache-ant installed 
Initial Steps: Go to the folder where you have extracted the Freecol sources 


Type ‘ant’. This will start building the Freecol application with debugging information (you will see as was 
in GDB, the -g flag is used by the compiler if you look at the build.xml ). 


This will take some minutes and you will have a new FreeCol.jar file 
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vac] 
javac] /usr/home/cnei ra/workshopj db/ freecol/src/net/sf/freecol/common/mode 
ava:400: warning: unmappable character for encoding ASCII 


* latitude. Thus, -30 equals 3677N, and 46 equals 4677S. 


avac | 
avac | 
J 


] 
} 
vac 


usr/home/cnei ra/workshopj db/freecol/src/net/st/freecol/common/mode 
ava:400: warning: unmappable acter for encoding ASCII 


* latitude. Thus, -3@ equals 307?7N, and 46 equals 4677S. 


fjusr/home/cneira/workshopjdb/free src/net/sf/freecol/common/mode 
ava:400: warning: unmappable character for encoding ASCII 


* latitude. Thus, <-30 equals 3677N, and 46 equals 46775 


ey ey Ea Se Gy Ga Se el | 


9 warnings 


fexec] Result: 1 


properties: 


[exec] Result: 


age: 


jar] Building jar: /usr/home/cnei ra/workshopj db/freecol 


D SUCCESSFUL 
lL time: 53 seconds 


There are two ways to start debugging using JDB. 


The first one is to give JDB the initial class (the one that has the main function) and start from there. As 
the JVM has not started, you must type ‘run’ to start the program. We will use the second approach that 
is to start a JVM and connect to it: 


$ java -jar -Xmx256M -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=6000 FreeCol.jar 
--no-intro 


Here, | use the --no-intro parameter because there is an issue with Freecol where the intro movie 
freezes the application. 


Then, we need to attach to this running JVM with the following command: 
$ jdb -attach localhost: 6000 

Now, let's start with some basic commands. 

Setting breakpoints : 


First, we need to know how to set breakpoints: 
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stop in <class-name>.<method-name> Sfop on entry to the given method. 
stop at <class-name>:<line-number> Sfop at the given line. 

clear <class-name>.<method-name> Remove the specified breakpoint. 
clear <class-name>:<line-number> Remove the specified breakpoint. 
Stepping in Stepping out 


When you hit a breakpoint, you either take a look at the data or continue executing the program. Here is 
the syntax for continuing, stepping, etc. 


cont 
As the name suggests, continue the execution after you hit the breakpoint. 
step 


Execute the current line . If the breakpoint is at a method call, it will stop the program at the method 
entry (same as GDB). 


next 


Execute the current line. If the breakpoint is at a method call, do not stop at the method entry (same as 
GDB). 


step up 
Execute until the current method returns to its caller (in GDB, the command is called finish). 
Take a look at the source code: 


Like in GDB, this command bears the same name. You need to execute jdb where the source is located 
or use the command in the JDB and provide the directory with the source code. 


list Lists 10 lines starting 4 before the current line. 

list <linenumber> Lists 10 lines starting 4 before the given line. 

list <method_name> Lists the first 10 lines of the given method. 

Taking a peek 

To look at the values of variables or expressions, we have the following commands: 
print <name> Prints the current value of the given variable. 

print <expression> Prints the value of the given expression. 


locals Prints the values of all variables local to the current method (In GDB, this one is called Info 
locals). 
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Calling for help 
The same as GDB, just type ‘help’. 
Option flags 


When you use the jdb command instead of the java command on the command line, the jdb command 
accepts similar options as those accepted by java command, including -D, -classpath, and -X options. 
The following list contains additional options that are accepted by the jdb command. 


-help 
Displays a help message. 
-sourcepath dir1:dir2:... 


Uses the specified path to search for source files in the specified path. If this option is not specified, then 
use the default path of dot (.). 


-attach address 

Attaches the debugger to a running JVM with the default connection mechanism. 
-listen address 

Waits for a running JVM to connect to the specified address with a standard connector. 
-launch 


Starts the debugged application immediately upon startup of JDB. The -launch option removes the need 
for the run command. The debugged application is launched and then stopped just before the initial 
application class is loaded. At that point, you can set any necessary breakpoints and use the cont 
command to continue execution. 


-listconnectors 

Lists the connectors available in JVM. 

-connect connector-name:name1=value1 

Connects to the target JVM with the named connector and listed argument values. 
-dbgtrace [flags] 

Prints information for debugging the jdb command. 

-tclient 


Runs the application in the Java HotSpot VM client. 
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-tserver 
Runs the application in the Java HotSpot VM server. 
-Joption 


Passes option to the JVM, where, option is one of the options described on the reference page for the 
Java application launcher. For example, -J-Xms48m sets the startup memory to 48 MB. 


Exercise: 
Set a breakpoint when a new Colony is built, and print the stacktrace 
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Introduction to The JDB 
Start Debugging 


JDB, like GDB, is a cli-debug tool that will help us find bugs in our code, hopefully with little effort. 

The application that will be used to demonstrate the usefulness of JDB will follow the setting as the 
last one. The application we are going to use is called Freecol. 

"The FreeCol team aims to create an Open-Source version of Colonization (released under the GPL). At 
first, we'll try to make an exact clone of Colonization. " 

Like the last time we used GDB, we need the Freecol application sources to be compiled with extra 
debugging symbols (-g flag to the Java compiler). 


Requirements : 


1- You must download the following package : 
http://prdownloads.sourceforge.net/freecol/freecol-0.10.7-src.zip?download 
2- You must have the openjdk7 package installed. 

3- You must have apache-ant installed 


Initial Steps: 


1- Go to the folder where you have extracted the Freecol sources. 

2- Type ’ant’. This will start building the Freecol application with debugging information (you will see as 
was in GDB, the -g flag being used by the compiler if you look at the build.xml). 

3- This will take some minutes and you will have a new FreeCol.jar file. 


Start debugging 

There are two ways to start debugging using JDB. 

The first one is to give JDB the initial class (the one that has the main function) and start from there. As 
the JVM has not started, you must type ‘run’ to start the program. We will use the second approach 


that is to start a JVM and connect to it: 


S$ java -jar -Xmx256M -Xdebug 


=SEUN TOW? Cealisport=di Socket; server=y,address=6000 BrecCol< jar —=no-1ntro 
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Here, | use the --no-intro parameter because there is an issue with Freecol where the intro movie 
freezes the application. 


Then, we need to connect it to the running JVM with the following command: 
$ jdb -attach localhost:6000 

Now, let's start with some basic commands. 

Setting breakpoints: 

First, we need to know how to set breakpoints: 


stop in <class-name>.<method-name> Stop on entry to the given method. 


stop at <class-name>:<line-number> Stop at the given line. 
clear <class-name>.<method-name> Remove the specified breakpoint. 
clear <class-name>:<line-number> Remove the specified breakpoint. 


Stepping in Stepping out 
When you hit a breakpoint, you either take a look at the data or continue executing the program. Here is 
the syntax for continuing, steping, etc.: 


cont 


As the name suggests, continue the execution after you hit the breakpoint. 
step 


Execute the current line . If the breakpoint is at a method call, it will stop the program at the method 
entry (same as GDB). 
next 


Execute the current line. If the breakpoint is at a method call, do not stop at the method entry (same as 
GDB). 
step up 


Execute until the current method returns to its caller (In GDB, the command is called finish). 
Take a look at the source code: 

Like in GDB, this command bears the same name. 

list Lists 10 lines starting 4 before the current line. 

list <linenumber> Lists 10 lines starting 4 before the given line. 


list <method_name> Lists the first 10 lines of the given method. 


Taking a peek 
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To look at the values of variables or expressions, we have the following commands: 


print <name> Prints the current value of the given variable. 

print <expression> Prints the value of the given expression. 

locals Prints the values of all variables local to the current method. 
Calling help 


The same as GDB, just type ‘help’. 
Here is a table of the command syntax between GDB and JDB. 


JDB commands GDB commands 
step Ss 
next n 
cont Cc 
stop in/at b 
clear info b 
step up finish 
up/down f 
where bt 
print/dump p 
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Working with Core 
Dumps in GDB 


What is a core dump? 


Acore dump or core file is produced when a signal (man signal) indicates that is not possible to continue 
executing the program because of an error (for example SIGSEV). A core file contains the current state 
of the program when it was executing (the stack of each thread, contents of the CPU registers, the 
values of global and static variables, etc..), and if we are lucky and the program that dumped core was 
compiled with debug options, the core file will have information about the source code and lines. A core 
file is used to do a postmortem analysis of what happened to our application. If we have the same 
versions of libraries between hosts, you could take a core produced, let’s say on a production machine, 
and analyze it in another machine that has the same libraries and versions (it is a must. Otherwise, your 
core file analysis will be invalid). 


Allowing to dump cores 


For the examples, we are going to use FreeBSD 10.0-STABLE to check if your installation is enabled to 
generate core files. You could execute the following commands as root: 


# sysctl -a |grep kern.coredump kern.coredump: 0 
# sysctl kern.coredump=1 kern.coredump: 0 -> 1 
# sysctl -a | grep kern.coredump kern.coredump: 1 


kern.coredump = 0 means that the core files will not be generated. Thus, use the sysctl (man sysctl ) 
facilities to change that, and make it permanent by editing /etc/sysctl.conf (man sysctl.conf). 


You could see the difference by just executing this simple program: 
#include<stdio.h> 

#include<stdlib.h> 

Pi Mean (). 1. abore() =] 


Just name it test.c or whatever name you want. Then, do a make test or make <the name you have 
chosen> and you will see it core dumping and leaving a test.core file or none if you did not apply the 
changes to the kern.coredump variable. The abort() causes the program to abnormally terminate hence 
generating a core file. 
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Working with the core file 
Now that we have a core file, to take a look at it and start debugging, we just type: 
% gdb test test.core 


% gdb <name of the executable must be in your path> <core file> 


ClLTAaA@icA! .~/ Work Op *® U 
NU gdb 6.1.1 [FreeBSD] 
opyright 2004 Free Software Foundation, Inc. 
DB is free software, covered by the GNU General Public License, and you are 
elcome to change it and/or distribute copies of it under certain conditions. 
ype “show copying” to see the conditions. 
here is absolutely no warranty for GDB. Type “show warranty” for details. 
his GDB was configured as “amd64-sarcel-freebsd"...(no debugging symbols found)... 


ore waS generated by ‘test’. 

rogram terminated with signal 6, Aborted. 

heading symbols from /lib/libc.so.7...(no debugging syabols found)...done. 
oaded symbols for /lib/libc.so.7 

heading symbols from /lib/Id-elf.so.1...(no debugging symbols found)...done. 
oaded symbols for /lib/ld-elf.so.1 

0 oe in kill © from /lib/libc.so.7 

ad 


As you see, there are no debugging symbols. | used the make command which took my CFLAGS from 
the 


/etc/make.conf file, and there, | did not have the —g flag set. 
Debugging without source code 


If debugging sometimes with the source code proves cumbersome, definitely, without the source code 
could be complicated. But let’s try debugging without it! 


We need to introduce the concept of Application Binary Interface. An ABI defines how system calls 
parameters are passed (which registers contain which argument), how functions are called and in which 
binary format information should be passed from one program component to another. FreeBSD ABI 
conforms to these specifications: 


http://www.x86-64.org/documentation/abi.pdf 


http:/Awww.sco.com/developers/devspecs/abi386-4.pdf 


Both documents are a good reading. You could also take a look at the developer’s handbook: 
http://www.nl.freebsd.org/doc/en/books/developers-handbook/x86-system-calls.html 


Now, maybe we are in a better position to debug some core files. Let’s try this simple program: 
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i =< > 
#include<stdlib.h> 


int calling_convention_test(int a,int b,int c, int d) 


printf("printing a: %d b : Sd c: Md d: Md\n",a ,b,c,d); 
return 999; 


int main() 


int res; 
res = calling_convention_test(1,2,3,4); 


abort(); 


I’m on x86_ 64 architecture. So, let’s see page 18, section 3.2.3 Parameter passing. In a nutshell, it says 
that integer parameters (CLASS TYPE INTEGERS) will be passed using registers 
%rdi, Yrsi, Yrdx, Y%rcx,%r8 and %r9. To check that, compile the program by typing: 


S cc <source.c> -o <exec name> 
Then, run the executable program that will get you a core file and load this into GDB as usual. 
S$ gdb <exec> <core file> 


Now, we will use the disassemble command (disas is the short form). This command dumps a range of 
memory as machine instructions. The default memory range is the function surrounding the program 
counter of the selected frame. Two arguments are interpreted as a range of memory to dump. By default, 


GDB uses the AT&T notation (http://en.wikipedia.org/wiki/X86_assembly_ language). If we want to 
disassemble a specific function, just type: 


(gdb) disas <function name> 
We will always have a main function, but what about the names of the other ones? You could type: 
(gdb) info func 


And you will get all function names. This works if the executable has not been stripped (man 1 strip). 
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ay gdb checkparameters checkparameters.core 
NU gdb 6. 1.1 rpreeBSD) 
opyright 2004 Free Software Foundation, Inc. 
DB is free software, covered by the GNU General Public License, and you are 
elcome to change it and/or distribute copies of it under certain conditions. 
ype “show copying” to see the conditions. 
here is absolutely no warranty for GDB. Type “show warranty” for details. 
his GDB was configured as “and64-marcel-freebsd”...(no debugging symbols found)... 
ore was generated by ‘checkparameters' 
rogram terminated with signal 6, Aborted. 
Reading syabols from /lib/libc.so.7...(no debugging symbols found)...done. 
oaded sysbols for /lib/libc.so.7 
eading symbols from /lib/1ld-elf.so.1...(no debugging symbols found)...done. 
-oaded syabols for /lib/ld-elf.so.1 
0x000000080095d0ea in kill Q froma /lib/libc.so.7 
gdb) bt 
0x00000008009Sd0ea in kill () from /lib/libc.so.7 
0x000000080095b819 in abort () from /lib/libc.so.7 
0x00000000004007aa in sain () 
(gdb) f 2 
0x00000000004007aa in main () 
(gdb) i lo 
No symbol table info available. 
gdb) i main 
ndefined info command: “main”. Try “help info”. 
gdb) disas sain 
luap of asseabler code for function sain: 
)x0000000000400780 <main+0>: push “rbp 
1x0000000000400781 <main+l1>: BOV Srsp,%rbp 
x0000000000400784 <main+4>: BOV $0x400816 , %edi 
)x0000000000400789 <main+9>: mov $0x1,%es1 
)x000000000040078e <main+14>: BOV $0x2 ,%edx 
x0000000000400793 <main+19>: BOV $0x3 ,%ecx 
x0000000000400798 <main+24>: sov $0x4 ,%r8d 
7x000000000040079e <main+30>: xor Seax ,%eax 
x00000000004007a0 <main+32>: callq Ox4004dc <printf@pit- 
x00000000004007a5 <main+37>: callq 0x40050c <abort@p/t> 
x00000000004007aa <main+42>: nopw Ox0(*rax,¥rax,1) 
nd of asseabler dusp. 
gdb) §j 


Figure 1. disas command on main 


J >a> i Nive ) e 
ae of assesbler code for function calling_ con ention_test: 
00000000004007b0 <calling_convention_test+0>: push %rbp 
DxO00 00000004007b1 <calling_convention_test+ l>: g0V Xrsp,¥rbp 
0x00000000004007 ecalling_convention_test+4>: sov %ecx ,%r8d 
' c Tling_convention_test¢7>: sov %edx ,¥ecx 
ing_convention_test+9>: sov %esi ,¥edx 
ing_convention_test¢1i>: gov %edi , %esi 
ing_convention_test+13>: BOV $0x400816 ,%edi 
ing_convention_test+18>: xor %eax , eax 
ing_convention_test+20>: callq Ox4004dc <printfépit- 
ling _convention_test+25>: HOV $0x3e?7 ,%eax 
ling_convention_test+30>: pop Srbp 
o 0000004007 cf ccalling_convention_test+31>: retq 
of assesbler dusp. 
) disas sain Ox04007cf 
f asseabler code from 0x400780 to 0x4007cf: 
<maineO>: push Srbdo 
<aainel>: B0V S&rsp,¥rbp 
0784 <aaine4>: Bov 30 x400816, 
789 <waineS-: 80v $0x1. ¥es 
, <aainel4>: B0ov $0 x2 Sedx 
793 <main+19> B0v $0x3 , ¥ecx 
} <maine24>: B0V $0x4 , ¥r8d 
<main430>: xor Bear , eax 
‘ 0 <main+32>: callq Ox4004dc <printfé@pit> 
000000004007a5 <main+37>: callq 0xé400S0c «<abortG6p!it 
Dx00000000004007aa «<main+42 nopw Ox0(%rax ,%rax,1) 
0x00000000004007b0 «<ca Ting. conv fention_test+0>: push &rbp 
x00000000004007b1 <calling_convention_test+ >: 80V Xrsp,¥rbp 
00000000004007b4 <calling_convention_test+4>: sov %ecx ,%r8d 
Dx00000000004007b7 <cal ling_convention_test+7>: gov %ecx , ¥ecx 
0 7b9 <calling_convention_vtest+9>: sov Kes! , ¥edx 
<calling_convention_test¢1i>: gov %edi , *est 
<cal ling_convention_test¢13>: qv $0x400816 ,%edi 
2 <cal ling convention _test+18>: xor Xeax , eax 
<cal ling_convention_test+20>: callq Ox4004de <printfépit 
9 <cal ling _convention_test+25>: BOV $0x3e7 , ¥eax 
DOO OO?ce <calling_convention_test+30>: pop %rbp 
nd of assesbler duap. 
(gdb) § 


Figure 2. disas command on calling_convention_test 


Set a breakpoint at the calling convention_test function, and run the program from the top. 


0 0x0000000000400784 in ca 


gdb) disas 


ing_convention_test 


uep of asseabler code for function calling_convention_test: 


x0000000000400780 
x0000000000400781 
x0000000000400/784 
x0000000000400/85 
x0000000000400790 
x0000000000400793 
x0000000000400796 
x0000000000400799 
x000000000040079c 
x000000000040079F 
x00000000004007a2 
x00000000004007a5 
x00000000004007a9 
x00000000004007 ac 
x00000000004007 ae 
x00000000004007b3 
x00000000004007b8 
x00000000004007bb 
x00000000004007bd 
x0000000000400/7c1 
x0000000000400/c2 
x00000000004007c3 


<Calling_convention_test+0>: 
<calling_convention_test+l>: sov 
<calling_convention_test+4>: sub 
<calling_convention_test+8>: lea 
<cal ling_convention_test+16>: 
<cal ling_convention_test+19>: 
<cal ling_convention_test+22>: 
<calling_convention_test+25>: 
<cal ling_convention_test+28>: 
<calling_convention_test+31>: 
<cal ling_convention_test+34>: 
<cal ling_convention_test+37>: 
<cal ling_convention_test+41>: 
<calling_convention_test+44>: 
<calling_convention_test+46>: 
<cal ling_convention_test+51>: 
<calling_convention_tese+56>: 
<calling_convention_test+59>: 
<calling_convention_test+61>: 
<cal ling_convention_test+65>: 
<calling_convention_test+66>: 
<cal ling_convention_test+67>: 


pus 


nd of assembler dump. 


gdb) x/d 0x400846 


x400846 <.rodata>: 


gdb) x/s 0x400846 


x400846 <.rodata>: 


0x600968 


0x0 
0x4 
0x3 
0x2 


0x1 


Ox7ffttttfesbo 
Ox7 fT fftttfesbo 


0x0 


Oxf TT TT800692bd970 -8 
0x80081c520 7 
0x246 
0x7 fff ffFFea30 
Ox7fffffffea4s 
Ox7 ff fff FFeazs 


0x1 


1852404336 
“printing a: db : &d c: 


293864 


62 
0 
4 
3 
2 


1 
Ox7fTfttfftfe obo 
Ox7TTttftfesbo 
0 

794328540 
34368242976 
582 
140737488349744 
140737488349768 
140737488349752 
1 


h Srbp 


Srsp,*rbp 
$0x20,%rsp 
0x400846 , Srax 


OV 
8OV 
OV 
BOV 
ROV 
ROV 
ROV 
B0V 
80V 
8OV 
callq 
nov 
nOV 
8OV 
add 
pop 
retg 
nopw 


Kd d: %d\n" 
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Sedi ,-0x4(%rbp) 
%esi ,-0x8(%rbp) 
%edx, -Oxc(%rbp) 
%ecx, -0x10(%rbp) 
-0x4(%rbp) , %es7 
~Ox8(%rbp) , %edx 
-Oxc(%rbp) , %ecx 
-0x10(%rbp) ,%r8d 
Srax,%rdi 
$0x0,%al 
0x4004dc <printf@p] t> 
$0x3e7 ,%ecx 

%eax ,-0x14(¥%rbp) 
%ecx ,¥%eax 
$0x20,%rsp 

“rbp 


%cs:0x0(%rax,%rax,1) 


Figure 3. Using disas command 


As you Can see, our parameters in the registers are being taken from the stack. To see this ina 
friendlier interface, use the layout command: 


(gdb) layout asm (gdb) layout regs 


That will split the screen showing the registers as they change at the top of the screen, and the asm 
instructions at the bottom. 
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group: genera 
0x600968 6293864 
0x4 4 
Ox 2 0x1 1 
Ox? FETT ET FeSO Ox7 fff tfttesdo sp OxT FFETEFTe9bo Ox7 Ff ft ff fesbo 
Ox 0 9 Oxf TT rTsoosrtadcéhs 
Ox246 52 


6 $87 
Ox? ff tft ffea4ss 


Oxi i 
Ox206 $18 
0x3 +9 
0 
0 


(Ton 
vention. te 
onvention_ test 


onvention_test > calle de <printf@p!t> 
vention. teste¢dl> v 0 , Secx 
t+ 0x14 (¥rbp) 


Figure 4. Seeing machine code and register info at the same time 


If you see the contents of the registers specified in the AMD64 ABI (“%rdi, Y%rsi, Yrdx, Yrcx,%r8 and %r9), 
you will see the contents of the arguments that are being passed to the function <main+9>. If you only 
want to check the registers without using the GUI, just type: 


(gdb) info regs 
Or just print the one you need to check, for example %rcx register: 


(gdb) p $rcex 


What about if the executable has been stripped? 


This is tougher to debug as we don't have the symbol tables available. The symbol tables have been 
stripped from the binary, which means, we do not have the function names, all references to the names 
are gone and we only have all the operations using memory addresses. That means, we cannot even 
set a breakpoint using main. However, all we need to know the address of our main function. 


Let’s take a look at another example program: 
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#include<string.h> 
#include<stdlib.h> 


void seemegodown(int, int); 
int sain() 


int a, b; 
seemegodown(a,b); 


} 


void seemegodown(int a, int b) 
I 
L 


int C 


Figure 5. Code example 2 


Compile it as usual, and strip it: 

= CC Seqgrauller.c. =o segraulier 
Ss Strip ségqtaulter 

Now, let’s find where the main is at. 


(gdb) info file 
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yabols froma “/usr/home/cneira/workshop/segfaulter”. 

reeBSD sultithreaded core duap file: 
‘/usr/home/cneira/workshop/segfaulter.core’', file type elf64-x86-64-freebsd. 
0x0000000000600000 - 0x0000000000601000 is loadi 
0x000000080061b000 - 0x000000080063b000 is load2 
0x000000080081b000 - 0x000000080081d000 is load3 
0x0000000800b8e000 - 0x0000000800b9a000 is load4 
0x0000000800b9a000 - 0x0000000800bc3000 is load5 
0x0000000800c00000 - 0x0000000801400000 is load6 
0x00007 fF fff ffd F000 0x00007FTTFFFFFOO0 is load7 
0x000000080081d190 - 0x00000008008221b4 is .hash in /lib/libc.so.7 
0x00000008008221b8 - 0x00000008008279c8 is .gnu.hash in /lib/libc.so.7 
0x00000008008279c8 - 0x00000008008399T8 is .dynsym in /lib/libc.so.7 
0x00000008008399F8 - 0x0000000800840890 is .dynstr in /lib/libc.so.7 
0x0000000800840890 - 0x0000000800842094 is .gnu.version in /lib/libc.so.7 
0x0000000800842098 - 0x0000000800842160 is .gnu.version_d in /lib/libc.so.7 
0x0000000800842160 - 0x000000080084e190 is .rela.dyn in /lib/libc.so.7 
0x000000080084e190 - 0x0000000800853308 is .rela.pit in /lib/libc.so.7 
0x0000000800853308 - 0x000000080085331b is .init in /lib/libc.so.7 
0x000000080085331c - 0x000000080085697c is .plt in /lib/libc.so.7 
0x0000000800856980 - 0x000000080095fb58 .text in /lib/libc.so.7 
0x000000080095fb58 - 0x000000080095 fb66 .fini in /lib/libc.so.7 
0x000000080095 Tb80 0x000000080096c8d4 .rodata in /lib/libc.so.7 
0x000000080096c8d4 - 0x0000000800972c08 is .eh_frame_hdr in /lib/libc.so.7 
0x0000000800972c08 - O0x000000080098d84c is .eh_frame in /lib/libc.so.7 
0x0000000800b8e000 - 0x0000000800b8e004 is .tdata in /lib/libc.so.7 
0x0000000800b8e010 0x0000000800b8e098 .tbss in /lib/libc.so.7 
0x0000000800b8e010 - 0x0000000800b8e030 is .ctors in /lib/libc.so.7 
0x0000000800b8e030 - O0x0000000800b8e048 is .dtors in /lib/libc.so.7 
0x0000000800b8e048 - 0x0000000800b8e050 ~jcr in /lib/libc.so.7 
0x0000000800b8e050 0x0000000800b92 f98 .data.rel.ro in /lib/libc.so.7 
0x0000000800b92f98 - 0x0000000800b93138 is .dynamic in /lib/libc.so.7 
0x0000000800b93138 - 0x0000000800b93a48 is .got in /lib/libc.so.7 
0x0000000800b93a48 - 0x0000000800b95588 is .got.pit in /lib/libc.so.7 
0x0000000800b95590 0x0000000800b991b0 .data in /lib/libc.so.7 
0x0000000800b991b0 - 0x0000000800bc2f08 is .bss in /lib/libc.so.7 
0x0000000800600158 - 0x0000000800600214 is .hash in /lib/ld-elf.so.1 
0x0000000800600218 - 0x00000008006002f4 is .gnu.hash in /lib/ld-elf.so.1 
0x00000008006002f8 0x0000000800600598 .dynsya in /lib/ld-elf.so.1 
0x0000000800600598 - 0x00000008006006d3 .dynstr in /lib/ld-elf.so.1 
0x00000008006006d4 - 0x000000080060070c -gnu.version in /lib/ld-elf.so.1 
0x0000000800600710 - 0x00000008006007d8 is .gnu.version_d in /lib/ld-elf.so.1 
0x00000008006007d8 0x0000000800602080 is .rela.dyn in /lib/Id-elf.so.1 
0x0000000800602080 - O0x00000008006155f9 is .text in /lib/ld-elf.so.1 
0x0000000800615600 - 0x000000080061776a is .rodata in /lib/ld-elf.so.1 
0x000000080061776c - 0x0000000800618180 is .eh_frame_hdr in /lib/ld-elf.so.1 
0x0000000800618180 - 0x000000080061ac58 is .eh_frame in /lib/ld-elf.so.1 
0x000000080081b000 - 0x000000080081b008 is .dtors in /lib/ld-elf.so.1 
0x000000080081b010 - 0x000000080081b318 is .data.rel.ro in /lib/ld-elf.so.1 
0x000000080081b318 - 0x000000080081b458 is .dynamic in /lib/Id-elf.so.1 
0x000000080081b458 - 0x000000080081b550 is .got in /lib/ld-elf.so.1 
0x000000080081b550 - 0x000000080081b568 is .got.plt in /lib/ld-elf.so.1 
0x000000080081b570 - O0x000000080081b9e0 is .data in /lib/ld-elf.so.1 
0x000000080081b9e0 - 0x000000080081c770 is .bss in /lib/ld-elf.so.1 

local exec file: 
‘/usr/home/cneira/workshop/segfaulter’, file type elf64-x86-64-freebsd. 
Entry point: 0x4004e0 
0x0000000000400200 - 0x0000000000400211 is .interp 
0x0000000000400214 - 0x0000000000400244 is .note.tag 
0x0000000000400248 - 0x0000000000400280 is .hash 
0x0000000000400280 - 0x00000000004002b0 is .gnu.hash 
0x00000000004002b0 - 0x0000000000400388 is .dynsys 
0x0000000000400388 - 0x00000000004003e0 is .dynstr 
0x00000000004003e0 0x00000000004003f2 is .gnu.version 
0x00000000004003F8 - 0x0000000000400418 is .gnu.version_r 


Figure 6. Info file command 


There is the Entry point: 0x4004e0 
Let’s set a breakpoint at that memory area: (gdb) b *0x4004e0 


Then, as we know that the program text area (code) is between 0x4004e0 and 0x4007a8. We could 
disassemble the range between those addresses (not really a thing you will want to do in a real 
application) other option that is nicer is just to display the program counter $pc the next couple of 
instructions every time you go down one instruction (nexti). 


(gdb) display /i Spc 


nusder J 
Ox4004e0 0x04007a8 


ler code froa 0x4004e0 to 0x4007a8: 


0 <exit@p)t+20>: 
Mel <exit@p]t+2 Lb 
4 <exit@p]t+24>: 
6 <exit@p]t+26>: 
3 <exit@p]t+28>: 


c <exite@plt+ 
exiteép!t+3 


<exi tép |) te3 
4 <exitép!t+40>: 
7 <exiv@p)t+43>: 
C <ex1TOp]t+48> 
<€x1 TOp1t+56o: 
6 <exit@p]t+58>: 
<ex1 tOp1t+65>: 
<exi tOp]t+69>: 
<exi t@pit+ 
exit@p|t+74 
cexi tap! t+ 
<¢xi té@plte : 
<exi t@p)t+82>: 
‘ <@x1 TOp)t+84>: 
<x1TOp1t+87>: 
a <exit@p]t+94>: 
<€x1 TOp1t+96>: 
f <exit@p]1t+99.: 


>> & 


i> > i> & 


<¢xi tép | teil 
<exit@pltelli-: 
0 <exit@plt+lLl6>: 
3 <exit@plt+119>: 
<@xit@plt+121>: 
00548 <exit@p]t+124>: 
0054d <exit@p]t+129>: 
<exi t@p]t+131L> 
exitep]t+136>: 
exitep!t+141-: 
exi tépit+144>: 
> <exi TOp)t+150>: 


xrbp 

xrsp, 

eris 

urlé 

arl3 

wrl2 

%rbx 

ar ax 

erdi .%ri4 
(%r14) .Sri1$ 


20057c(%rip) # 0x600a80 
Md <exit@p1t+65> 
00$73(%rip) # 0x600a80 
“rl4 
%r15d,%r1$d 
0x400$3b <exit@p]t+1l1> 
(%r14) ,Srax 
Xrax, ¥rax 
0x400523 <exit@p)t+87> 
0x40053b <exit@plt+i1i> 


# 0x600a60 <__prognase> 


urax 


(Srax) ,%cl 
$0x2f ,.%c] 


<ext t@p1t+96> 
8. %eax 
3 trax 
Ox400S4f <exit@plt+13b> 
arsi .%rai 
0x40049c «<atexit@p]t> 
0x400$54 <exit@p)t+136> 
Ox4004ac <_init_tls@p 
$0x6008b8 , eax 
%rax,%rax 
0x400641 <exitépit+3 
$0x400660 , edi 


Figure 7. disas command addresses range 
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<exiteplt+6139: 
<e€xit@p1t+620>: 
<€xi t@p11+623>: 
<exi t@p1t+626>: 
<exit@p1t+631>: 
<€xi t@p11+636>: 
<exi t@p1t+640>: 
<exit@p1t+641>: 
<e€xi t@p1t+642>: 
<exi t@p1t+644>: 
<e€xi t@p11t+645>: 
<exit@p1t+648>: 
<exi t@p1t+652>: 
<exi t@p1t+657>: 
7 <e€xit@p1t+667>: 
<e€xi t@p1t+671>: 
<exi t@p1t+674>: 
<€xi t@p1t+677>: 
<e€xi t@p1t+680>: 
<exi t@p1t+683>: 
<€xi T@p11+686>: 
<exi t@p1t+689>: 
<exi t@p1t+691>: 
<exi t@p1t+696>: 
8 <exitép!t+700>: 
o ria . 


p 
‘rsp, %rbp 
$0x10 ,%rsp 
-0x4(%rbp) ,%edi 
-0x8(%rbp) ,%esi 
0x400750 <exit@p]1t+644> 
$0x0 , eax 
$0x10 ,%rsp 
%rbp 


Sax , %ax 

%rbp 

%rsp,%rbp 

$0x10 ,%rsp 
$0x48 , %eax 
$0x400 , %rdx 
-Oxc(%rbp) ,%rcx 
sedi , -0x4(%rbp) 
%esi ,-0x8(%rbp) 
-0x4(%rbp) ,%esi 
-0x8(%rbp) ,%esi 
%esi,-Oxc(%rbp) 
Srcx,%rdi 
%eax,%esi 
0x4004bc <menmset@p]t> 
$0x10 ,%rsp 

%rbp 


Here is our function (notice the call to memset in there) <exit@plt+644>. 
Useful commands for looking at machine code: 


Apart for the ones described earlier, you can use these ones to take a look at what is happening in your 
program. 


Examining memory 


Use the x command to examine memory. The syntax for the x command is x/FMT ADDRESS. The FMT 
field is a count followed by a format letter and a size letter. Use the help command ‘help x' to see them 
all. The ADDRESS argument can either be a symbol name, such as a variable, or a memory address. 


Examine the variable as a string: 

(gdb) x/s <var> 

Examine the variable as a character: 
(gdb) x/c <var> 

Examine the variable as 4 characters: 
(gdb) x/4c <var> 

Examine the first 32 bits of the variable: 
(gdb) x/t <var> 


Examine the first 24 bytes of the variable in hex: 
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(gdb) (sx -<var> 
Print the instructions at the memory area specified (gdb) x/i  <addr> 
Stepping 


As we inspect the machine code, we don’t have the source code. The commands that we usually use: 
next (n) and step(s) have their equivalents named with the ‘I’ suffix (for instruction): 


nexti (ni) 


stepi (si) 


Question: 


Based on the documentation, describe the GDB session steps that you will use to check the parameters 
being passed to the memset function, and the user-defined function of the program on example program 
2: 
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Introduction to 
Dtrace 


DTrace, or dynamic tracing, was first available in the Solaris 10 3/05 around 2005. DTrace is now 
available in FreeBSD beginning from 7.1 and Mac OS X from 10.5 (Leopard). DTrace differs from 
traditional tools in that code is instrumented dynamically (that means you can peek at the program 
without recompiling). 


From the handbook: https://www.freebsd.org/doc/en_US.1SO8859- 1/books/handbook/dtrace.html 


“The FreeBSD implementation provides full support for kernel DTrace and experimental support for 
userland DTrace. Userland DTrace allows users to perform function boundary tracing for userland 
programs using the pid provider, and to insert static probes into userland programs for later tracing. 
Some ports, such as databases/postgres-server and lang/php5 have a DTrace option to enable static 
probes. FreeBSD 10.0-RELEASE has reasonably good userland DTrace support, but it is not considered 
production ready. In particular, it is possible to crash traced programs.” 


Requirements 


I’m running FreeBSD 10.0-STABLE where Dtrace is already available as a kernel module. Typing the 
following as root will let you know that you are ready to fire up some probes using dtrace: 


(‘t:~ % sudo dtrace -1 | we -1l 


neira@Next:~ % 


If this fails, you need to recompile your kernel and follow the instructions from the handbook in here: 
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dtrace-enable.html 


Why do | care about DTrace? 


If you want to understand what is happening in your software without needing recompiled special 
versions of your applications (lots of debug messages, maybe recompile with debug flags to use a 
debugger?) and also centralize all your current instrumentation tools into just one, then you should care 
about DTrace. 


Some Features of DTrace. 


DTrace is dynamic: probes are enabled only when you need them. 
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No code is present for inactive probes. 

There is no performance degradation when you are not using DTrace. 

When the dtrace command exits, all probes are disabled and instrumentation removed. 
The system is reverted to its original state. 

DTrace is nondestructive. The system is not paused or quiesced. 

DTrace is designed to be efficient. No extra data are ever traced. 


Because of its safety and efficiency, DTrace can be used in production to solve real problems in real 
time. 


Predicates: A logical predicate mechanism allows actions to be taken only when user- specified 
conditions are met. Unwanted data is discarded at the source—never retained, copied, or stored. 


A high-level control language: DTrace is equipped with an expressive C-like scripting language known as 
D . It supports all ANSI C operators, which may be familiar to you and reduce your learning curve, and 
allows access to the kernel’s variables and native types. D offers user-defined variables, including global 
variables, thread-local variables, and associative arrays, and it supports pointer dereferencing. This, 
coupled with the runtime safety mechanisms of DTrace, makes it admirable 


Trying DTrace : 
We will try a default script that comes with our FreeBSD installation. Go to 


/usr/share/dtrace/toolkit and execute the script called procsystime. This script “only processes system 
Call time details.” Notice the only. 


oot@bsd: /usr/share/dtrace/toolkit # ./procsystime 
racing... Hit Ctrl-C to end... 
Cc 


lapsed Times for all processes, 


SYSCALL TIME (ns) 
Sigreturn 5778 
sigaction 12857 

fstat 15745 
getpid 18541 
__sysctl 39645 
munmap 47318 
getsockopt 51057 
mmap 58251 

read 97469 
sigprocmask 278332 
ioctl 474094 
clock_gettime 683811 
write 1123755 
_umtx_op 3011901084 
select 3014173473 


ootébsd: /usr/share/dtrace/toolkit # J 
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This one is pretty handy right away. Imagine what you could do with some imagination. Let’s see this 
one liner: which processes are executing the most system calls? 


oot@bsd:/usr/share/dtrace/toolkit # dtrace -n ‘syscall:::entry { @[pid, execname] = count();}' 
itrace: description ‘syscall:::entry ' matched 536 probes 


1398 preload 
1377 sendmail 
2118 sshd 
2273 dtrace 
oot@bsd: /usr/share/dtrace/toolkit # § 


Pretty impressive, isn't it? At this moment, you are wondering how all these years you have lived without 
DTrace. Using truss, strace ,lsof even gdb now seems pretty lame. Well, gdb is not so lame now. 


DTrace Scripting: 


DTrace scripts are written in the D language. You could take a look at this reference 
http://dlang.org/spec.html. 


Now, let’s write our first probe. A DTrace script has the following structure: 
Your probes 

/ predicate (usually you will create a filter here) / 

{ 

What are you going to do when you hit a probe 

} 


Let’s create a simple one to get used to the syntax, and later, we will dissect it line by line, ok? This one 
does not have a predicate, so it will capture all that the probe is asking for. 


A predicate is a conditional statement (IF statement if you like). 


syscall:::ent =» your 
ry probe 
(@[pid,execname] = > your 


| count(): action 


49 


Save this to a file called example1.d . Then, execute the script by typing: dtrace —s example1.d 
The probe section has the following syntax: 


provider :module: function: name 
What every section means: 


Provider The name of the DTrace provider that is publishing this 
probe. The provider name typically corresponds to the 
name of the DTrace kernel module that performs the 
instrumentation to enable the probe. 


Module If this probe corresponds to a specific program location, 
the name of the module in which the probe is located. 
This name is either the name of a kernel module or the 
name of a user library. 


Function 2 
If this probe corresponds to a specific program location, 
the name of the program function in which the probe is 
located. 

Name The final component of the probe name is a name that 


gives you some idea of the probe’s semantic meaning, 
such as BEGIN or ENDS. In this case, the probe says that 
is the entry of a function call. 


What providers are available to us in FreeBSD? Well, you should dig in and see what you need. 


root@bsd:"/dtracescripts # dtrace -1 i head -1 
ID PROVIDER MODULE FUNCTION NAME 
root@bsd:~/dtracescripts # dtrace -1 i head -Z 
ID PROVIDER MODULE FUNCTION NAME 
1 dtrace BEGIN 
root@bsd:"/dtracescripts # dtrace -1 i head -5 
ID PROVIDER MODULE FUNCTION NAME 
dtrace BEGIN 
dtrace END 
dtrace ERROR 


fbt kernel camstatusentrycomp entry 
# dtrace -! 


PROVIDER MUDULE FUNCTION NAME 
dtrace BEGIN 
dtrace END 
dtrace ERROR 

fbt kernel camstatusentrycomp entry 
fFbt kernel camstatusentrycomp return 
fbt kernel cam_compat_handle_Ox17? entry 


Now to this line: 


@[pid,execname] = count(); 
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This is called an aggregation, and is denoted by the @ special character. Aggregations are global in your 
Dtrace scripts. The syntax of an aggregation is as indicated below: 


@name[ keys 


aggfunc ( args 


Name: The name you choose for the aggregation. 


Keys: Comma-separated list of D expressions (in this case, we are asking for pid and the name of 
executable which triggers the probe). 


Aggfunc: This is one of the DIrace aggregating functions. Moreover, args is a comma-separated list of 
arguments appropriate for the aggregating function. 


Here are the aggregation functions available: 


Function 
Name 


count 
sum 
avg 
min 


max 


Iquantize 


quantize 


Arguments 


none 
scalar expression 
scalar expression 
scalar expression 


scalar expression 


scalar 
expression, 
lower bound, 
upper bound, 
step value 


scalar expression 


Result 


The number of times called. 

The total value of the specified expressions. 

The arithmetic average of the specified expressions. 
The smallest value among the specified expressions. 
The largest value among the specified expressions. 


A linear frequency distribution, sized by the specified 
range, of the values of the specified expressions. 
Increments the value in the highest bucket that is less 
than the specified expression. 


A power-of-two frequency distribution of the values of 
the specified expressions. Increments the value in the 
highest power-of-two bucket that is less than the 
specified expression. 
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Now, let’s add a predicate to the same script. If you looked at the output of the script you would have 
seen that it counted the system calls done by dtrace itself. Let’s filter that. 


oot@bsd:”/dtracescripts # dtrace -s examplei.d 
dtrace: script ‘examplel1.d’ matched 536 probes 
C 


1466 
1244 
1464 
1467 
1376 
1397 
1466 
1467 
1465 
1467 
1465 
1383 cron 

‘oot@bsd:”/vdtracescripts # cat examplel.d 

; sentry 
execname t= “dtrace” 


sh 
syslogd 
cron 
preload 
sendmail 
preload 
dd 

sh 

cron 
vmstat 
sh 
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@Cpid,execname] = count(): 


But how did | know that execname contained the name of the program being executed? Well, it is a 
built-in variable in DTrace. Here is a list of some Dtrace Built-in variables: 


DTrace Built-in Variables 
Type and Name Description 


The first ten input arguments to a probe represented as raw 64-bit integers. If 


int64_t argO, fewer than ten arguments are passed to the current probe, the remaining 
.., arg9 variables return zero. 
The typed arguments to the current probe, if any. The args|] array is accessed 
using an integer index, but each element is defined to be the type 
args|] corresponding to the given probe argument. For example, if args[] is referenced 
g by a read(2) system call probe, args[0] is of type int, args[1] is of type void *, 
and args[2] is of type size_t. 
uintptr_t The program counter location of the current thread just before entering the 
caller current probe. 
The name of the current working directory of the process associated with the 
string cwd current thread. 
The enabled probe ID (EPID) for the current probe. This integer uniquely 
uint_t epid identifiers a particular probe that is enabled with a specific predicate and set of 
- actions. 
int errno The error value returned by the last system call executed by this thread. 
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computations. 


uid_t uid The real user ID of the current process. 

uint64 t The current thread's saved user-mode register values at probe firing time. Use 
uregs|] of the uregs|]. 

uint64 t The current thread's active virtual machine register values at probe firing time. 
vmregs|] Use of the vmregsJ]. 


The current value of a nanosecond timestamp counter that is virtualized to the 
amount of time that the current thread has been running on a CPU, minus the 


uint64_t time spent in DTrace predicates and actions. This counter increments from an 

vtimestamp arbitrary point in the past and should only be used for relative time 
computations. 

uint64 t The current number of nanoseconds since 00:00 Universal Coordinated Time, 

walltimestamp January 1, 1970. 


You could take a look at the full listing in this URL: 
http://docs.oracle.com/cd/E18752_01/html/819-5488/gcfpz.html 


Assignment: 


Pick any program that you are certain that uses the strcmp(3) C library function, and print the arguments 
passed to this function. Tip: first create a script, similar to the example, to check for the strcmp call 
being used in any the programs executing in your machine. 
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